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ABSTRACT 

There  are  many  sources  of  data  available  to  enhance  commanders’  situational  awareness. 
Whether  the  mission  is  war  fighting,  stability  operations,  or  the  defense  of  the  homeland, 
decisions  must  be  made  based  on  the  information  available.  Information  comes  from  many 
varied  sources,  in  many  formats,  with  a  continuum  of  validity,  and  is  presented  in  a  variety  of 
ways.  Situational  awareness  and  decision  superiority  opportunity  is  increased  when  relevant 
data  is  effectively  presented  in  a  timely  manner  and  with  a  measure  of  confidence.  Exact, 
precise  data  is  easier  to  present  than  uncertain  data;  but  often  uncertain  data  may  be  enough 
for  a  decision,  based  on  rules  of  engagement  and  the  value  of  the  action  to  the  mission.  We 
have  developed  visualization  techniques  that  focus  on  uncertainty  for  the  purpose  of 
providing  decision-quality  command  and  control  information  for  both  ground-based  and 
airborne  applications.  A  key  focus  of  our  work  is  to  manage  the  visualization  techniques  so 
they  do  not  clutter  the  display,  as  a  cluttered  display  can  actually  decrease  situational 
awareness  by  masking  relevant  information. 

Data  can  be  presented  on  a  2D  or  3D  visual  display.  This  paper  proposes  a  method  for 
managing  2D  visualizations  to  increase  situational  awareness. 

Keywords:  Interaction  Techniques,  Command  and  Control,  Uncertainty,  Movement 
Prediction 

INTRODUCTION 

In  order  to  wage  a  successful  campaign,  next  generation  commanders  and  air  controllers  will 
need  to  assimilate  a  tremendous  amount  of  available  information,  from  a  wide  variety  of 
sources  and  types  ranging  from  hardcopy  HUMINT  reports  to  real-time  sensor  feeds.  Armed 
with  this  data  they  will  be  expected  to  see  through  the  "fog  of  information"  and  make  time- 
critical  decisions  and  to  quantify  the  effects  of  those  decisions,  all  in  the  face  of  uncertainty 
and  risk.  It  is  a  daunting  task  for  which  systems  and  tools  available  to  current  command 
centers  and  airborne  control  platforms  offer  no  proven  solution.  It  is  inarguable  that  decision 
support  systems  for  any  domain  are  dramatically  improved  by  the  addition  of  some  degree  of 
visualization.  To  date,  however,  even  the  most  state-of-the-art  visualization  approaches 
concern  themselves  primarily  with  visualization  of  data;  e.g.,  terrain,  maps,  iconic  depictions 
of  threat  lay-downs  and  flight  packages,  etc.  But  data  is  not  information,  and  it  is  information 
that  the  commander  needs  to  gain  the  decision  superiority  required  to  successfully  wage  his 
campaign  or  for  the  airborne  controller  to  gain  the  situational  awareness  needed  to  attain 
dominance  of  the  airspace. 

Much  of  Command,  Control,  Communications,  Computers  and  Intelligence  (C4I)  deals  with 
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making  decisions  in  the  face  of  uncertain  or  inadequate  data.  “Uncertainty  can  be  generally 
defined  as  an  inability  to  determine  a  variable  value  or  system  state  (or  nature)  or  to  predict  its 
future  evolution.”  [CBP,  p.  249]  Reducing  uncertainty  (fog  of  war)  and  in  the  process 
increasing  confidence  in  the  validity  of  the  information  increases  situational  awareness. 
“There  are  four  sources  of  ‘fog’  commanders  and  staffs  must  overcome  to  achieve  accurate 
situational  understanding:  inadequate  or  poor-quality  information;  misinterpretation  of 
information;  conflicting  information  or  choices;  and  too  much  information”  [FM].  Many 
visualization  techniques  are  developed  in  isolation  to  address  only  one  or  two  sources  of  fog. 
A  visualization  approach  is  required  in  which  individual  techniques  work  together  in  a  unified 
fashion  to  address  all  four  sources  of  fog. 

In  the  following  sections,  we  categorize  the  types  of  threat  location  and  movement  data  and 
discuss  possible  visual  representations.  We  will  explain  how  an  effective  interface  can 
prevent  these  representations  from  becoming  a  cluttered  and  unusable  visual  barrage  of  data, 
and  how  our  visualization  tools  can  better  interact  in  a  complex  system.  We  conclude  with  a 
discussion  of  related  products 

VISUALIZATION  GOALS 

The  goal  of  visualization  tools  is  not  to  fuse  data,  improve  estimates,  collaborate  multiple 
information  sources,  etc.  These  are  crucial  tasks,  and  the  visualization  tools  must  accept  and 
accommodate  the  data  from  these  steps,  but  the  goal  of  visualization  is  to  display  the  data  in 
such  a  way  as  to  aid  decision  makers  and  to  increase  their  situational  awareness.  According 
to  the  NATO  Code  of  Best  Practice,  “assessments  should  be  judged  by  their  ability  to  reduce 
uncertainty”  [CBP,  p.  252],  and  this  is  also  a  very  good  metric  by  which  to  judge  decision 
support  visualizations. 

Visualization  is  often  a  quick  method  to  communicate  information.  “Communication  is  about 
giving  the  receiver  of  a  message  a  right  impression,  not  about  formulating  a  statement  that  is 
formally  correct  on  its  own.”  [CBP,  p.  262]  “Visualization  techniques  will  be  helpful  in  this 
regard  since  they  are  usually  more  powerful  than  verbal  reference  to  abstract  concepts”  [CBP, 
p.  263].  This  will  prove  even  more  true  in  collaborative  coalition  environments  where 
participants  may  have  different  native  languages.  In  Alberts  and  Hayes,  collaboration  is  one 
of  the  key  dimensions  of  flexibility  for  situational  awareness.  [AH,  p.  148] 

One  technique  for  reducing  uncertainty  is  to  provide  visualization  tools  for  managing  the  data. 
Tools  that  solve  one  source  of  fog  at  the  expense  of  another  are  inadequate.  Our  techniques 
work  together  in  a  unified  fashion  to  address  all  four  sources  of  "fog".  A  key  focus  of  our 
work  is  to  manage  the  visualization  techniques  so  that  they  do  not  clutter  the  display,  and  thus 
in  turn  contribute  to  the  fog  of  war  through  the  fourth  source  "too  much  information". 

Operators  are  overwhelmed  with  command  and  control  information;  their  screens  becoming 
quickly  cluttered,  potentially  obscuring  important  data.  The  system  goal  is  to  provide 
advanced,  multi-sensory,  display  management  concepts  and  algorithms  for  visualizing 
command  and  control  information  in  simulated  environments.  Not  all  display  management 
techniques  are  appropriate  at  all  times.  MAK’s  visualization  toolkits  are  designed  as  a  set  of 
display  techniques.  From  this  set,  the  appropriate  subset  of  tools  can  be  chosen  based  on 
criteria  such  as  the  situation,  role  of  the  user,  or  even  the  personal  decision-making  techniques 
of  the  commander  or  staff.  These  tools  can  be  used  to  build  flexible,  customizable  displays. 
Five  key  techniques  were  considered: 
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•  Tailoring  the  Display.  Based  on  the  situation,  user’s  role,  individual  user 
preferences  and  screen  size  (hand-held,  desktop  or  data  wall). 

•  De-cluttering.  Focus  on  important  information  by  providing  the  ability  to  filter  out 
detail  unnecessary  to  the  task  at  hand.  Techniques  interoperate  with  each  other  in  a 
dynamic  environment. 

•  Integration.  Fluidly  change  visualization  techniques  as  the  situation  evolves. 

•  Cross  tool  control.  All  visualization  tools  can  show  alternate  views  of  the  same 
situation,  or  views  can  be  set  and  controlled  by  one  for  the  others. 

•  Use  of  Toolkit.  Easy  to  extend  techniques  and  customized  displays  to  support  future 
efforts. 

ORGANIZATION 
Knowledge  Organization 

We  can  coarsely  categorize  each  piece  of  the  battle  space  into  three  types:  areas  where  we 
have  no  presence  and  hence  no  data  collection  capability,  areas  where  we  have  collection 
coverage  and  no  threat  data  is  reported,  and  areas  where  we  have  coverage  and  have  collected 
threat  data.  Identification  of  each  of  these  areas  provides  different  benefits. 

One  of  the  most  extreme  examples  of  "inadequate  information"  is  no  information  at  all,  or  a 
knowledge  void.  By  simply  and  easily  identifying  regions  of  the  battle  space  where  we  do  not 
have  any  collection  capability,  commanders  can  make  more  informed  decisions  about  where 
to  put  emphasis  in  collection  plans. 

The  second  category,  areas  where  we  have  coverage  but  have  not  collected  threat  data,  is  also 
useful  to  identify.  These  areas  are  potentially  good  places  to  plan  supply  and  evacuation 
routes,  or  to  covertly  maneuver  friendly  forces  or  stage  into  strategic  positions.  Conversely, 
the  reason  for  lack  of  threat  data  in  this  region  may  be  the  result  of  inadequate  collection 
coverage.  Visualization  can  provide  a  confidence  factor  on  the  coverage  to  provide 
commanders  with  a  measure  of  validity. 

The  third  category,  areas  where  threat  data  has  been  collected,  is  the  most  prone  to  clutter. 
Collected  data  may  be  inadequate  or  conflicting.  Commanders  spend  most  of  their  time 
dealing  with  this  data  so  we  provide  interactive  techniques  and  tools  for  the  operator  to 
manipulate  this  data.  The  operator  can  use  the  tools  in  such  a  way  as  to  filter  out  non-impact 
data,  present  relevant  information,  apply  a  confidence  value  to  the  data  and  then  display  that 
data  to  decision  makers. 

Visual  Organization 

Overlays  correspond  to  traditional  acetate  sheets  overlaying  a  terrain  map,  upon  which  one 
can  draw.  Any  symbol  can  be  placed  on  an  overlay  layer.  Through  the  overlay  manager, 
individual  overlays  can  be  dynamically  created,  toggled  on  or  off,  or  even  reordered. 

For  a  user  specified  area  of  interest,  we  generate  a  2D  overlay  that  indicates  the  areas  where 
there  is  collection  coverage.  We  will  also  generate  the  inverse;  an  overlay  that  marks  the 
areas  where  we  have  no  data  from  any  source,  i.e.  the  knowledge  voids  (this  is  not  fully 
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implemented).  We  can  further  divide  the  area  of  coverage  to  show  areas  where  threat  reports 
have  been  received,  and  those  where  they  have  not. 

The  areas  of  coverage  on  an  overlay  may  be  drawn  as  semi-transparent,  or  with  a  variety  of 
patterned  fills  such  as  crosshatches  or  stripes.  Unlike  solidly  filled  blocks,  this  allows  details 
of  the  battle  space  to  remain  available  to  the  commander.  From  a  pragmatic  point  of  view, 
transparencies  provide  a  better  user  interface,  but  patterned  fills  are  faster  to  render  on  a 
display  (since  they  do  not  require  alpha  blending).  The  decision  of  which  to  use  will  be  based 
on  the  number  and  size  of  the  objects  displayed,  and  frame  update  requirements,  ffence 
allowing  the  user  to  choose  the  display  technique  is  important  to  overall  system  usability. 

Coverage  areas  may  be  drawn  in  any  color.  Although  standard  defaults  are  easiest  to 
recognize  and  use,  color  changes  may  be  necessary  to  provide  contrast  against  different 
terrain  map  backgrounds. 

Further  refinements  would  include  the  ability  to  distinguish  areas  of  coverage  by  source,  such 
as  sensor  type.  Each  source  type  could  be  placed  on  a  separate  overlay  and/or  encoded  with 
different  colors.  This  would  allow  the  user  to  interactively  toggle  each  one  (or  its  inverse)  on 
or  off,  and  to  see  them  individually  or  in  groups.  This  could  easily  display  the  coverage  of 
individual  systems  such  as  AWACS,  JSTARS,  COMPASS  CALL,  RIVET  JOINT,  U-2, 
UAVs,  PATRIOT  batteries,  or  to  see  all  air  defense  coverage  and  assets  or  all  passive 
collection  assets.  By  placing  the  display  of  these  overlays  under  user  control,  we  avoid 
cluttering  the  battlefield  map  when  the  data  is  not  useful,  more  specifically;  we  avoid  the 
fourth  problem  "too  much  information". 


■ 

Overlay  Manager 
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Available  Overlays 

■ 

■ 

:  0  AIR  DEFENSE 

j  -  MIM -104  Patriot 
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!-"f>u2 

rO  COMPASS  CALL 
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W  Allow  Overlay  Edit 

New  Layer  | 

L 

A 

Figure  1  The  overlay  manager  (right)  is  used  to  turn  on  both  air  defense  and  Intel  collection  overlay 
layers.  With  all  data  shown  at  the  same  time,  the  screen  (left)  is  so  cluttered  that  it  is  difficult  to 
understand  the  situation. 
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Figure  2  With  only  the  Air  Defense  overlay  layers  shown;  it  is  easier  to  comprehend  the  situation 


Figure  3  With  the  Air  Defense  layers  turned  off,  the  Intel  collection  becomes  more  apparent. 


DATA  UNCERTAINTY 

Verification  and  Consolidation  of  Threat  Data 

There  are  many  sources  of  data  available  to  commanders.  Another  source  of  uncertainty 
arises  from  the  quality  and  consistency  of  these  data  sources. 

Information  collected  may  be  inadequate  (saw  something,  but  not  sure  of  the  number  or  type), 
conflict  with  one  another  (if  multiple  units  have  different  view  points,  they  may  each  see 
something  different,  none  of  which  may  be  entirely  accurate),  be  misleading  due  to  temporal 
discrepancies  (two  reports  at  different  times  of  the  same  area),  simply  be  too  old,  or  be 
corrupt. 
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Data  corruption  may  be  accidental  through  transmission  failure  or  intentional  through  hostile 
acts,  where  the  enemy  actively  intercepts,  alters  or  destroys  data.  The  first  step  in  resolving 
the  uncertainty  is  to  be  able  to  organize  and  access  the  data  in  a  meaningful  way.  Two  types 
of  organization  are  time  and  space.  Creating  threat  icons  in  the  locations  specified  in  reports, 
the  commander  gains  a  spatial  overview  of  the  data.  Providing  the  operator  the  capability  to 
apply  a  collection  time  stamp  and  the  ability  to  provide  a  confidence  value  allows  the  decision 
maker  to  quickly  form  an  impression  of  the  timeliness  and  validity  of  the  data. 

There  may  still  be  ambiguity  and  uncertainty,  since  multiple  sightings  will  result  in  multiple 
threat  reports,  and  there  is  no  way  to  distinguish  between  two  friendly  units  sighting  the  same 
threat  unit  and  have  the  same  friendly  units  sight  different  opposing  force  units.  One  method 
for  dealing  with  the  data  quality  is  to  provide  extra  information,  but  structured  in  such  a  way 
that  it  does  not  overwhelm  the  display.  A  user  selects  an  area  of  interest  (AOI)  within  the 
battle  space  to  query  the  collection  sources.  A  list  is  displayed,  showing  how  the  information 
was  collected  and  where  and  when  the  collection  took  place.  For  example,  multiple  tanks 
could  have  seen  that  sub-area,  as  well  as  a  UAV.  To  assist  decision  makers,  again,  a  time 
stamp  and  confidence  value  can  be  attached  to  the  icon. 

Visual  Representation  of  Source  Data 

A  side  panel  shows  additional  pertinent  data  for  selected  list  items.  For  example,  report  time, 
externally  determined  confidence  ratings  of  the  source  data,  frequency  (for  sensors),  or 
external  forces  that  could  affect  the  accuracy  of  the  report  such  as  weather.  A  "show  view" 
button  could  pop  up  an  inset  window  to  show  what  that  entity  "saw"  from  that  entities 
viewpoint.  If  available,  UAV  video  or  still  images  from  any  source  could  be  inserted. 

Having  the  tools  to  display  selected  data,  apply  a  confidence  value,  apply  a  collection  time 
stamp,  and  manage  the  data  on  the  screen  allows  the  operator  to  better  present  the  situation. 
Based  on  this  enhanced  human  understanding  of  the  battle  space,  the  operator  can  use  our 
visualization  tools  to  interact  with  the  system  and  clarify  the  display  for  decision  makers. 

One  such  action  available  is  to  move  objects  from  the  2D  display  to  a  “lost  contact”  list.  This 
action  is  appropriate  when  one  report  indicates  an  enemy  presence  in  an  area,  but  a  later 
report  does  not  see  that  unit.  In  this  case  we  know  the  unit  has  moved,  so  we  do  not  want  to 
show  the  icon  in  that  area,  but  we  certainly  do  not  want  to  forget  about  that  opposing  force 
unit. 

Another  visualization  capability  is  designed  to  indicate  areas  covered  within  a  certain  time 
period.  For  example,  although  we  don't  have  a  current  collection  asset  in  a  certain  part  of  the 
battle  space,  if  a  sensor  covered  the  area  previously  or  will  cover  the  area  later,  a  time  stamp 
on  the  collection  track  or  area  would  inform  the  decision  maker  of  the  area  collection  status. 
Visualization  tools  are  provided  to  allow  commanders  to  see  at  a  glance,  and  be  able  to  filter, 
various  collection  and  weapon  system  coverage,  sensor  tracks,  and  airspace  coordination 
measures. 

INTERACTION  OF  VISUALIZATION  TECHNIQUES 

The  goal  of  the  interface  is  to  simultaneously  provide  access  to  all  of  the  types  of  data  and 
provide  the  operator  tools  to  enhance  the  value  of  the  information  and  prevent  the  volume  of 
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data  from  overwhelming  the  decision  maker.  Thus,  we  propose  the  following  list  as 
suggestions  for  enhancing  the  display: 

•  One  source  type  per  overlay,  plus  a  universal  overlay  switch  which,  for  convenience, 
turns  all  overlays  on  or  off  as  a  block. 

•  Display  sensor  coverage  for  a  period  of  time  specified  in  a  user  modifiable 
configuration  file.  Each  type  of  sensor  may  have  a  different  time  period  specified. 

For  example,  days  for  satellites,  hours  for  planes,  last  hour  for  DI.  This  will  also 
allow  the  user  to  configure  for  different  types  of  combat. 

•  Allow  the  user  to  set  the  period  of  time  on  a  slider,  from  the  maximum  specified  to 
immediate.  That  way  the  user  can  dynamically  determine  the  coverage  over  the  time 
period  of  interest. 

•  Allow  coverage  segments  to  be  created  based  on  altitude  intervals.  For  example, 
coverage  under  1000  ft. 

•  Terrain  projection.  Project  sensor  coverage  on  to  the  terrain.  Clearer  picture  of  areas 
that  have  been  covered. 

•  Allow  the  operator  the  capability  to  apply  a  time  stamp  and  confidence  value  to  any 
displayed  icon. 

TOOL  INTERACTION 

MAK  has  3  visualization  tools:  the  MAK  Plan  View  Display  (2D  tactical  map  display),  the 
MAK  Stealth  (traditional,  realistic  3D  viewer),  and  the  MAK  StealthXR  (3D  viewer  which 
also  displays  non-realistic  data  such  as  2525B  icons,  NCBR  (nuclear,  chemical,  biological, 
radiological)  hazards,  satellite  imagery,  threat  domes,  etc). 

These  three  visualization  products  inter-operate  and  are  commercially  available  and  supported 
on  a  variety  of  platforms.  The  StealthXR  is  a  new  product  that  has  recently  been  launched. 
The  PVD  and  Stealth  are  mature  products.  All  products  natively  support  HFA  and  DIS,  while 
plug-in  APIs  can  be  tailored  to  import  entity  locations  and  tracks  from  various  C4I  protocols. 
All  products  support  a  variety  of  terrain  formats,  raster  maps  and  vector  data.  All  products 
have  been  run  on  desktops  and  large  screens.  To  date,  only  the  2D  PVD  has  been  run  on  a 
hand-held  device,  although  we  see  no  barriers  to  supporting  3D  on  a  hand-held. 


RESULTS  AND  FUTURE  WORK 

The  USAF  contract  under  which  this  work  has  been  done  is  still  not  completed.  The  work 
will  be  integrated  into  our  products,  and  made  available  as  part  of  a  commercial  off  the  shelf 
(COTS)  toolkit.  As  such,  this  paper  refers  to  a  commercial  product  whose  on-going 
development  and  maintenance  will  continue  for  many  years.  Many  of  our  plans  are  based  on 
preliminary  feedback  from  subject  matter  experts  and  trade  show  demonstrations,  but  will 
change  with  additional  feedback  (such  as  that  hopefully  generated  by  this  paper).  Current 
plans  include: 

•  Saving  individual  preferences  for  visualization  techniques,  such  as  which  overlays  are 
by  default  on  or  off  etc. 

•  Implementing  a  variety  of  techniques  to  distinguish  confidence  levels  for  individual 
data  points. 
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•  Allowing  filtering  by  confidence  levels.  For  example,  show  data  reported  by  sources 
with  reliability  ratings  of  70%  or  higher. 

•  Implementing  the  inverse  coverage  display  (knowledge  voids). 

•  Adding  some  of  the  proposed  visualization  interaction  techniques. 
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